{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using Multiple Metrics in Environments \n",
    "\n",
    "This notebook will go over how to record multiple metrics with HyperparameterHunter, how to interpret the results, and how to switch between them for hyperparameter optimization.\n",
    "\n",
    "As with most examples, we will start with preparing our data.\n",
    "\n",
    "# 1. Format DataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from sklearn.datasets import load_breast_cancer\n",
    "\n",
    "data = load_breast_cancer()\n",
    "train_df = pd.DataFrame(data.data, columns=[_.replace(\" \", \"_\") for _ in data.feature_names])\n",
    "train_df[\"diagnosis\"] = data.target"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Set Up Environment\n",
    "\n",
    "Now we'll set up our `Environment`. If you've gone through the other examples, everything below should be pretty standard, except for the `metrics`. In most examples, we give `metrics` a single metric to record, but what if we just can't choose? Answer: Give `Environment` a bunch of metrics in `metrics`! Notice that we provide the individual metrics in a few different formats accepted and documented by `Environment`. \n",
    "\n",
    "First, near the top, we import `f1_score` from `sklearn.metrics`. Continuing to our `metrics`... \n",
    "1. We start with the string \"roc_auc_score\", identifying the `sklearn.metrics` callable, and we name it **\"roc_auc\"**\n",
    "2. We add our imported `f1_score`, and name it **\"f1\"**\n",
    "3. We customize `f1_score` to use the `average=\"micro\"` kwarg, and we name it **\"f1_micro\"**, and\n",
    "4. We customize `f1_score` again, using the `average=\"macro\"` kwarg this time, and we name it **\"f1_macro\"**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cross-Experiment Key:   'rc2_ba0MazcGWpIULNzkv6KJMCrskIiTwpjNCPnyei4='\n"
     ]
    }
   ],
   "source": [
    "from hyperparameter_hunter import Environment, CVExperiment\n",
    "from sklearn.metrics import f1_score\n",
    "\n",
    "env = Environment(\n",
    "    train_dataset=train_df,\n",
    "    results_path=\"HyperparameterHunterAssets\",\n",
    "    target_column=\"diagnosis\",\n",
    "    metrics=dict(\n",
    "        roc_auc=\"roc_auc_score\",\n",
    "        f1=f1_score,\n",
    "        f1_micro=lambda y_true, y_pred: f1_score(y_true, y_pred, average=\"micro\"),\n",
    "        f1_macro=lambda y_true, y_pred: f1_score(y_true, y_pred, average=\"macro\"),\n",
    "    ),\n",
    "    cv_type=\"KFold\",\n",
    "    cv_params=dict(n_splits=10, shuffle=True, random_state=42),\n",
    "    verbose=1,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n",
    "Now, any Experiments we execute will record all four of these metrics!\n",
    "\n",
    "# 3. Perform Experiments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<14:52:55> Validated Environment:  'rc2_ba0MazcGWpIULNzkv6KJMCrskIiTwpjNCPnyei4='\n",
      "<14:52:55> Initialized Experiment: '392b7a26-04f5-4c0f-a4d8-f80867392d60'\n",
      "<14:52:56> Hyperparameter Key:     '0EL40_MPuHYTdL_IS7vHZS8MavkZZvnkUI-vAb46XZY='\n",
      "<14:52:56> \n",
      "<14:52:56> \n",
      "<14:52:56> FINAL:    OOF(roc_auc=0.95858, f1=0.97230, f1_micro=0.96485, f1_macro=0.96211)  |  Time Elapsed: 0.79187 s\n",
      "<14:52:56> \n",
      "<14:52:56> Saving results for Experiment: '392b7a26-04f5-4c0f-a4d8-f80867392d60'\n"
     ]
    }
   ],
   "source": [
    "from lightgbm import LGBMClassifier\n",
    "\n",
    "experiment_0 = CVExperiment(\n",
    "    model_initializer=LGBMClassifier,\n",
    "    model_init_params=dict(\n",
    "        boosting_type=\"gbdt\", max_depth=-1, min_child_samples=5, subsample=0.5, verbose=-1,\n",
    "    ),\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n",
    "As we can see above, the final report for `experiment_0` shows all four metrics, each with different values.\n",
    "\n",
    "You may be wondering what happens when we perform hyperparameter optimization. Which of our metrics will be optimized?\n",
    "An excellent question! The answer is, the first metric - unless we tell our optimizer otherwise. An example will better illustrate this.\n",
    "\n",
    "# 4. Hyperparameter Optimization\n",
    "\n",
    "We'll start by setting aside a `model_init_params` dict, so we can easily reuse them later. That's all - nothing sneaky going on there!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validated Environment with key: \"rc2_ba0MazcGWpIULNzkv6KJMCrskIiTwpjNCPnyei4=\"\n",
      "\u001b[31mSaved Result Files\u001b[0m\n",
      "\u001b[31m_______________________________________________________________________________________\u001b[0m\n",
      " Step |       ID |   Time |      Value |   boosting_type |   num_leaves |   subsample | \n",
      "Experiments matching cross-experiment key/algorithm: 1\n",
      "Experiments fitting in the given space: 1\n",
      "Experiments matching current guidelines: 1\n",
      "    0 | 392b7a26 | 00m00s | \u001b[35m   0.95858\u001b[0m | \u001b[32m           gbdt\u001b[0m | \u001b[32m          31\u001b[0m | \u001b[32m     0.5000\u001b[0m | \n",
      "\u001b[31mHyperparameter Optimization\u001b[0m\n",
      "\u001b[31m_______________________________________________________________________________________\u001b[0m\n",
      " Step |       ID |   Time |      Value |   boosting_type |   num_leaves |   subsample | \n",
      "    1 | 59d3494e | 00m00s | \u001b[35m   0.96941\u001b[0m | \u001b[32m           dart\u001b[0m | \u001b[32m          26\u001b[0m | \u001b[32m     0.5665\u001b[0m | \n",
      "    2 | 9da4b557 | 00m01s |    0.96705 |            dart |           37 |      0.6449 | \n",
      "Optimization loop completed in 0:00:01.855642\n",
      "Best score was 0.9694122932191745 from Experiment \"59d3494e-d6da-4646-abbb-8173bcf62936\"\n"
     ]
    }
   ],
   "source": [
    "from hyperparameter_hunter import BayesianOptPro, Real, Integer, Categorical\n",
    "\n",
    "OPT_MODEL_INIT_PARAMS = dict(\n",
    "    boosting_type=Categorical([\"gbdt\", \"dart\"]), \n",
    "    num_leaves=Integer(15, 45), \n",
    "    max_depth=-1, \n",
    "    min_child_samples=5, \n",
    "    subsample=Real(0.4, 0.7),\n",
    "    verbose=-1,\n",
    ")\n",
    "\n",
    "optimizer_0 = BayesianOptPro(iterations=2, random_state=32)\n",
    "optimizer_0.forge_experiment(LGBMClassifier, OPT_MODEL_INIT_PARAMS)\n",
    "optimizer_0.go()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n",
    "Now, take note of the single saved experiment that was found by `optimizer_0`. It lists the experiment ID given to the `experiment_0` we performed above. Furthermore, `optimizer_0`, lists the value of `experiment_0` as 0.95858. Therefore, we know that `optimizer_0` is using \"roc_auc\" score as its `target_metric` to optimize, because that is the final \"roc_auc\" value reported by `experiment_0`.\n",
    "\n",
    "# 5. Changing Target Metrics\n",
    "\n",
    "Suppose we now want to perform additional rounds of `BayesianOptPro` using our \"f1_micro\" metric as the optimized `target_metric`, instead. We would need to start all over from scratch, right? WRONG! HyperparameterHunter recorded all four of the metrics we declared in `env` for all experiments executed during optimization, as well! \n",
    "\n",
    "Even better, telling HyperparameterHunter to switch `target_metric`s is easy! Here's how to do it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validated Environment with key: \"rc2_ba0MazcGWpIULNzkv6KJMCrskIiTwpjNCPnyei4=\"\n",
      "\u001b[31mSaved Result Files\u001b[0m\n",
      "\u001b[31m_______________________________________________________________________________________\u001b[0m\n",
      " Step |       ID |   Time |      Value |   boosting_type |   num_leaves |   subsample | \n",
      "Experiments matching cross-experiment key/algorithm: 3\n",
      "Experiments fitting in the given space: 3\n",
      "Experiments matching current guidelines: 3\n",
      "    0 | 392b7a26 | 00m00s | \u001b[35m   0.96485\u001b[0m | \u001b[32m           gbdt\u001b[0m | \u001b[32m          31\u001b[0m | \u001b[32m     0.5000\u001b[0m | \n",
      "    1 | 9da4b557 | 00m00s | \u001b[35m   0.97188\u001b[0m | \u001b[32m           dart\u001b[0m | \u001b[32m          37\u001b[0m | \u001b[32m     0.6449\u001b[0m | \n",
      "    2 | 59d3494e | 00m00s | \u001b[35m   0.97364\u001b[0m | \u001b[32m           dart\u001b[0m | \u001b[32m          26\u001b[0m | \u001b[32m     0.5665\u001b[0m | \n",
      "\u001b[31mHyperparameter Optimization\u001b[0m\n",
      "\u001b[31m_______________________________________________________________________________________\u001b[0m\n",
      " Step |       ID |   Time |      Value |   boosting_type |   num_leaves |   subsample | \n",
      "    3 | 2e0ab6d1 | 00m01s |    0.96661 |            gbdt |           43 |      0.5827 | \n",
      "    4 | 5396fb5a | 00m00s |    0.96837 |            dart |           18 |      0.5036 | \n",
      "Optimization loop completed in 0:00:01.705806\n",
      "Best score was 0.968365553602812 from Experiment \"5396fb5a-9f10-4ff9-9141-0361d2c192db\"\n"
     ]
    }
   ],
   "source": [
    "optimizer_1 = BayesianOptPro(target_metric=\"f1_micro\", iterations=2, random_state=32)\n",
    "optimizer_1.forge_experiment(LGBMClassifier, OPT_MODEL_INIT_PARAMS)\n",
    "optimizer_1.go()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n",
    "The only difference between the code for `optimizer_1` and the code for `optimizer_0` before is the addition of `target_metric=\"f1_micro\"`.\n",
    "\n",
    "That's all we have to do! Notice that, once again, we see `experiment_0` at the top of the saved experiments being learned from, and now it shows a value of 0.96485. With a quick scroll upwards, we can verify that is the \"f1_micro\" score originally reported by `experiment_0`.\n",
    "\n",
    "We can also see two other saved experiments that were located, which are the two experiments produced by `optimizer_0`. Note that their values also differ from those reported by `optimizer_0`, because `target_metric=\"f1_micro\"` now, instead of the inferred \"roc_auc\" default.\n",
    "\n",
    "# 6. I Can't Make Up My Mind\n",
    "\n",
    "What if we now decide that we actually want to optimize using our normal \"f1\" metric, instead of either \"roc_auc\" or \"f1_micro\"? Easy!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validated Environment with key: \"rc2_ba0MazcGWpIULNzkv6KJMCrskIiTwpjNCPnyei4=\"\n",
      "\u001b[31mSaved Result Files\u001b[0m\n",
      "\u001b[31m_______________________________________________________________________________________\u001b[0m\n",
      " Step |       ID |   Time |      Value |   boosting_type |   num_leaves |   subsample | \n",
      "Experiments matching cross-experiment key/algorithm: 5\n",
      "Experiments fitting in the given space: 5\n",
      "Experiments matching current guidelines: 5\n",
      "    0 | 392b7a26 | 00m00s | \u001b[35m   0.97230\u001b[0m | \u001b[32m           gbdt\u001b[0m | \u001b[32m          31\u001b[0m | \u001b[32m     0.5000\u001b[0m | \n",
      "    1 | 2e0ab6d1 | 00m00s | \u001b[35m   0.97379\u001b[0m | \u001b[32m           gbdt\u001b[0m | \u001b[32m          43\u001b[0m | \u001b[32m     0.5827\u001b[0m | \n",
      "    2 | 5396fb5a | 00m00s | \u001b[35m   0.97507\u001b[0m | \u001b[32m           dart\u001b[0m | \u001b[32m          18\u001b[0m | \u001b[32m     0.5036\u001b[0m | \n",
      "    3 | 9da4b557 | 00m00s | \u001b[35m   0.97778\u001b[0m | \u001b[32m           dart\u001b[0m | \u001b[32m          37\u001b[0m | \u001b[32m     0.6449\u001b[0m | \n",
      "    4 | 59d3494e | 00m00s | \u001b[35m   0.97914\u001b[0m | \u001b[32m           dart\u001b[0m | \u001b[32m          26\u001b[0m | \u001b[32m     0.5665\u001b[0m | \n",
      "\u001b[31mHyperparameter Optimization\u001b[0m\n",
      "\u001b[31m_______________________________________________________________________________________\u001b[0m\n",
      " Step |       ID |   Time |      Value |   boosting_type |   num_leaves |   subsample | \n",
      "    5 | 00d4afe9 | 00m00s |    0.97772 |            dart |           28 |      0.5654 | \n",
      "    6 | 5af92081 | 00m00s |    0.97500 |            dart |           33 |      0.4150 | \n",
      "Optimization loop completed in 0:00:01.783474\n",
      "Best score was 0.977715877437326 from Experiment \"00d4afe9-23d8-4e20-9a66-d07241924455\"\n"
     ]
    }
   ],
   "source": [
    "optimizer_2 = BayesianOptPro(target_metric=\"f1\", iterations=2, random_state=32)\n",
    "optimizer_2.forge_experiment(LGBMClassifier, OPT_MODEL_INIT_PARAMS)\n",
    "optimizer_2.go()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "Just like that, `optimizer_2` is reporting our \"f1\" scores! Let's finish by optimizing with the last of our four metrics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validated Environment with key: \"rc2_ba0MazcGWpIULNzkv6KJMCrskIiTwpjNCPnyei4=\"\n",
      "\u001b[31mSaved Result Files\u001b[0m\n",
      "\u001b[31m_______________________________________________________________________________________\u001b[0m\n",
      " Step |       ID |   Time |      Value |   boosting_type |   num_leaves |   subsample | \n",
      "Experiments matching cross-experiment key/algorithm: 7\n",
      "Experiments fitting in the given space: 7\n",
      "Experiments matching current guidelines: 7\n",
      "    0 | 392b7a26 | 00m00s | \u001b[35m   0.96211\u001b[0m | \u001b[32m           gbdt\u001b[0m | \u001b[32m          31\u001b[0m | \u001b[32m     0.5000\u001b[0m | \n",
      "    1 | 2e0ab6d1 | 00m00s | \u001b[35m   0.96389\u001b[0m | \u001b[32m           gbdt\u001b[0m | \u001b[32m          43\u001b[0m | \u001b[32m     0.5827\u001b[0m | \n",
      "    2 | 5396fb5a | 00m00s | \u001b[35m   0.96590\u001b[0m | \u001b[32m           dart\u001b[0m | \u001b[32m          18\u001b[0m | \u001b[32m     0.5036\u001b[0m | \n",
      "    3 | 5af92081 | 00m00s | \u001b[35m   0.96597\u001b[0m | \u001b[32m           dart\u001b[0m | \u001b[32m          33\u001b[0m | \u001b[32m     0.4150\u001b[0m | \n",
      "    4 | 9da4b557 | 00m00s | \u001b[35m   0.96975\u001b[0m | \u001b[32m           dart\u001b[0m | \u001b[32m          37\u001b[0m | \u001b[32m     0.6449\u001b[0m | \n",
      "    5 | 00d4afe9 | 00m00s | \u001b[35m   0.96981\u001b[0m | \u001b[32m           dart\u001b[0m | \u001b[32m          28\u001b[0m | \u001b[32m     0.5654\u001b[0m | \n",
      "    6 | 59d3494e | 00m00s | \u001b[35m   0.97167\u001b[0m | \u001b[32m           dart\u001b[0m | \u001b[32m          26\u001b[0m | \u001b[32m     0.5665\u001b[0m | \n",
      "\u001b[31mHyperparameter Optimization\u001b[0m\n",
      "\u001b[31m_______________________________________________________________________________________\u001b[0m\n",
      " Step |       ID |   Time |      Value |   boosting_type |   num_leaves |   subsample | \n",
      "    7 | b795e6b5 | 00m01s |    0.96411 |            dart |           38 |      0.6733 | \n",
      "    8 | 75c7a723 | 00m01s |    0.96590 |            gbdt |           42 |      0.5383 | \n",
      "Optimization loop completed in 0:00:02.065230\n",
      "Best score was 0.9659000106541658 from Experiment \"75c7a723-3e61-4d41-97f8-6cda477dcc3c\"\n"
     ]
    }
   ],
   "source": [
    "optimizer_3 = BayesianOptPro(target_metric=\"f1_macro\", iterations=2, random_state=32)\n",
    "optimizer_3.forge_experiment(LGBMClassifier, OPT_MODEL_INIT_PARAMS)\n",
    "optimizer_3.go()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 7. Bonus Exercises\n",
    "\n",
    "If you've been reading the documentation as you should be, you may have noticed the `target_metric` argument of all children of `BaseOptPro` is usually a tuple. The `BayesianOptPro` class we used above is just one of the descendants of `BaseOptPro`, but we were passing `target_metric` values of strings.\n",
    "\n",
    "As the documentation notes, all `target_metric` values are cast to tuples, in which the first value identifies which dataset's evaluations should be used. The default behavior is to target the \"oof\", or out-of-fold, predictions' results. So, when we were using `target_metric=\"<string>\"` in our examples above, our optimizer interpreted it as `target_metric=(\"oof\", \"<string>\")`.\n",
    "\n",
    "This allows us to tell our optimizers to optimize metrics calculated using predictions on other datasets, like a holdout dataset.\n",
    "For example, had we initialized `Environment` with a `holdout_dataset`, our experiments would actually calculate 8 metrics instead of the 4 they currently do: 4 for our OOF predictions, and 4 for our holdout predictions. Then, if we wanted to optimize using holdout evaluations, we can use `target_metric=(\"holdout\", <metric_name>)`."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
